Papers with annotation cost
Reducing Confusion in Active Learning for Part-Of-Speech Tagging (2021.tacl-1)
Copied to clipboard
| Challenge: | Existing algorithms for annotating parts of speech are not optimal for all languages. |
| Approach: | They propose to use a data selection algorithm to select useful training samples to minimize annotation cost. |
| Outcome: | The proposed strategy outperforms existing strategies on six typologically diverse languages. |
Automatic Generation of Contrast Sets from Scene Graphs: Probing the Compositional Consistency of GQA (2021.naacl-main)
Copied to clipboard
| Challenge: | Recent studies show that supervised models exploit data artifacts to achieve good test scores while their performance severely degrades on samples outside their training distribution. |
| Approach: | They propose a method which automatically generates contrast sets for the visual question answering task by using a semantic input representation. |
| Outcome: | The proposed method computes the answer of perturbed questions, thus reducing annotation cost and enabling thorough evaluation of models’ performance on various semantic aspects. |
Generative Data Augmentation for Aspect Sentiment Quad Prediction (2023.starsem-1)
Copied to clipboard
| Challenge: | Existing approaches to analyze text contain rewrites and inconsistency between text and quads. |
| Approach: | They propose a new approach to analyze aspect terms, opinion terms, sentiment polarity in text . they augment quads and train a quads-to-text model to generate corresponding texts . |
| Outcome: | The proposed method outperforms existing methods and achieves state-of-the-art performance on two datasets. |
FreeTransfer-X: Safe and Label-Free Cross-Lingual Transfer from Off-the-Shelf Models (2022.findings-naacl)
Copied to clipboard
| Challenge: | Existing work on cross-lingual transfer has not studied how to leverage knowledge of rich-resource languages without labels. |
| Approach: | They propose a 2-step knowledge distillation framework to achieve knowledge transfer from off-the-shelf models in rich-resource languages. |
| Outcome: | The proposed method reduces annotation cost and protects private labels. |
Bridge-Based Active Domain Adaptation for Aspect Term Extraction (2021.acl-long)
Copied to clipboard
| Challenge: | Existing methods to transfer aspect terms are limited because they require labeled pivot words or expensive computing resources. |
| Approach: | They propose a method that actively supplements transferable knowledge by recognizing syntactic roles as pivots instead of links to pivots. |
| Outcome: | The proposed method significantly outperforms existing methods. |
Active Learning for Multilingual Semantic Parser (2023.findings-eacl)
Copied to clipboard
| Challenge: | Existing multilingual semantic parsing datasets are limited in translation effort due to data imbalance. |
| Approach: | They propose a first active learning procedure for multilingual semantic parsing (AL-MSP) it selects only a subset from existing datasets to be translated, they propose . |
| Outcome: | The proposed method significantly reduces translation costs with ideal selection methods. |
CAL-Log: Cost-Aware Active Learning with Logarithmic Cognitive Effort Modeling and Online Adaptation to Human Annotation Behavior (2026.acl-srw)
Copied to clipboard
| Challenge: | Standard uncertainty sampling assumes that annotating a 500-word document requires the same effort as a 50-word tweet, leading to suboptimal resource allocation when documents vary in length. |
| Approach: | They propose a cost-aware AL variant using logarithmic cost modeling where C(x) is the predicted annotation time for document x and L(x), is its token length. |
| Outcome: | Experiments on ten text classification benchmarks show a 3.3 speedup over BADGE and 3.9 over Entropy sampling to reach F1=0.80, with large effect sizes. |
Meta Self-Refinement for Robust Learning with Weak Supervision (2023.eacl-main)
Copied to clipboard
| Challenge: | Recent methods leverage self-training to build noise-resistant models . however, the teacher trained under weak supervision may have fitted a substantial amount of noise and therefore produce incorrect pseudo-labels. |
| Approach: | They propose a framework that encourages teacher to refine its pseudo-labels to effectively combat label noise from weak supervision. |
| Outcome: | The proposed framework outperforms state-of-the-art methods by 11.4% in accuracy and 9.26% in F1 score on eight NLP benchmarks. |
Multi-dimensional Evaluation of Empathetic Dialogue Responses (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Prior efforts to measure conversational empathy focus on expressed communicative intents, but ignore the fact that conversation is also a collaboration involving both speakers and listeners. |
| Approach: | They propose a multi-dimensional empathy evaluation framework to measure both expressed intents from the speaker’s perspective and perceived empathy from the listener’s viewpoint. |
| Outcome: | The proposed framework measures both expressed intents from the speaker’s perspective and perceived empathy from the listener’s viewpoint. |
Neural Ranking with Weak Supervision for Open-Domain Question Answering : A Survey (2023.findings-eacl)
Copied to clipboard
| Challenge: | Neural ranking models require substantial amounts of relevance annotations, which is costly to scale. |
| Approach: | They propose to train a NR model with weak supervision instead of annotations . they use a structured overview of standard WS signals used for training a model . |
| Outcome: | The proposed approach reduces the cost of annotations by using weak supervision instead of a parametric model. |
Pre-trained Language Model Based Active Learning for Sentence Matching (2020.coling-main)
Copied to clipboard
| Challenge: | Existing active learning approaches for natural language processing ignore the characteristics of natural language. |
| Approach: | They propose a pre-trained language model based active learning approach for sentence matching that provides linguistic criteria to measure instances and help select more effective instances for annotation. |
| Outcome: | The proposed approach can achieve greater accuracy with fewer labeled training instances. |
DialogueCSE: Dialogue-based Contrastive Learning of Sentence Embeddings (2021.emnlp-main)
Copied to clipboard
| Challenge: | Conventional approaches to learning sentence embeddings from dialogues employ the siamese-network for this task, but such architecture yields a large gap between training and evaluating. |
| Approach: | They propose a dialogue-based contrastive learning approach to learn sentence embeddings from dialogues using a siamese-network. |
| Outcome: | The proposed model outperforms baseline methods on three multi-turn dialogue datasets in terms of MAP and Spearman’s correlation measures. |
Progressive Class Semantic Matching for Semi-supervised Text Classification (2022.naacl-main)
Copied to clipboard
| Challenge: | Recent semi-supervised learning methods have achieved impressive performance . semi-controlled learning can be used to reduce the annotation cost of text classifiers . |
| Approach: | They propose a semi-supervised learning process that builds a standard K-way classifier and a matching network for the input text and the Class Semantic Representation (CSR). |
| Outcome: | The proposed method improves baselines and overall is more stable. |
Learning a Cost-Effective Annotation Policy for Question Answering (2020.emnlp-main)
Copied to clipboard
| Challenge: | State-of-the-art question answering systems require large amounts of training data for which labeling is time consuming and thus expensive. |
| Approach: | They propose a framework for annotating QA datasets that entails learning a cost-effective annotation policy and a semi-supervised annotation scheme. |
| Outcome: | The proposed approach can reduce up to 21.1% of the annotation cost compared with traditional methods . the proposed approach is based on a cost-effective annotation policy and semi-supervised annotation scheme . |
Context-aware Information-theoretic Causal De-biasing for Interactive Sequence Labeling (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing deep learning models for sequence labeling are expensive and time-consuming. |
| Approach: | They propose an interactive sequence labeling that allows training directly with the user feedback . they identify context and feedback biases by formulating interactive sequence labels via a Structural Causal Model. |
| Outcome: | The proposed approach can effectively alleviate the biases and can be learnt with the user feedback. |
ActiveEA: Active Learning for Neural Entity Alignment (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to combining knowledge Graphs (KGs) are incomplete but complementary to each other. |
| Approach: | They propose a novel Active Learning framework for neural EA that creates highly informative seed alignments to obtain more effective models with less annotation cost. |
| Outcome: | The proposed framework significantly improves sampling quality with good generality across different datasets, EA models and amount of bachelors. |
Adaptive Multi-Task Transfer Learning for Chinese Word Segmentation in Medical Text (C18-1)
Copied to clipboard
| Challenge: | Chinese word segmentation (CWS) tools face a performance drop when dealing with domain text . domain-specific CWS requires extremely high annotation cost due to ambiguity caused by domain terms and writing style . |
| Approach: | They propose to exploit domain-invariant knowledge from high resource to low resource domains to build Chinese word segmentation models. |
| Outcome: | The proposed model achieves higher accuracy than single-task CWS and other transfer learning baselines . the model is based on domain-invariant knowledge from high resource to low resource domains based in the biomedical domain . |
Active Learning with Non-Uniform Costs for African Natural Language Processing (2026.findings-eacl)
Copied to clipboard
| Challenge: | Annotating datasets for African languages is challenging due to the continent's vast linguistic diversity, complicating development of NLP systems. |
| Approach: | They propose a cost-aware active learning method that integrates BatchBALD acquisition strategy with a 0-1 Knapsack optimization objective to select informative and budget-efficient samples. |
| Outcome: | The proposed method outperforms BALD, BatchBALD, and stochastic sampling variants across cost scenarios on the MasakhaNEWS multilingual news classification benchmark covering 11 African languages. |
Incremental Learning from Scratch for Task-Oriented Dialogue Systems (P19-1)
Copied to clipboard
| Challenge: | Existing task-oriented dialogue systems cannot guarantee that all user needs are taken into account in the design phase. |
| Approach: | They propose a new incremental learning framework to design task-oriented dialogue systems without pre-defining user needs. |
| Outcome: | The proposed framework is robust to unconsidered user actions and can update itself online with less annotation cost. |
An Experimental Design Framework for Label-Efficient Supervised Finetuning of Large Language Models (2024.findings-acl)
Copied to clipboard
Gantavya Bhatt, Yifang Chen, Arnav Das, Jifan Zhang, Sang Truong, Stephen Mussmann, Yinglun Zhu, Jeff Bilmes, Simon Du, Kevin Jamieson, Jordan Ash, Robert Nowak
| Challenge: | Supervised finetuning (SFT) on instruction datasets has shown immense potential in improving the zero-shot generalization capabilities observed in large language models (LLMs). |
| Approach: | They propose to use experimental design to minimize the computational cost of active learning by identifying useful subsets of samples to annotate from an unlabeled pool. |
| Outcome: | The proposed methods save 50% of the annotation cost compared to random sampling on generative tasks. |
Pre-train or Annotate? Domain Adaptation with a Constrained Budget (2021.emnlp-main)
Copied to clipboard
| Challenge: | Recent work shows that pre-training in-domain language models can boost performance when adapting to a new domain. |
| Approach: | They propose to combine annotation and pre-training to maximize performance under budget constraints. |
| Outcome: | The proposed approach is based on the annotation cost of three procedural text datasets and pre-training cost of 3 in-domain language models. |
A Survey of Active Learning for Natural Language Processing (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing literature surveys on active learning for NLP are too specific or too general, covering deep active learning. |
| Approach: | They propose to use active learning to improve model learning and annotation cost for NLP problems. |
| Outcome: | The proposed approach is based on a large dataset of data-driven machine learning models. |
Less is More: Attention Supervision with Counterfactuals for Text Classification (2020.emnlp-main)
Copied to clipboard
| Challenge: | Specifically, we explore the advantage of counterfactual reasoning, over associative reasoning . Adding human supervision to attention has been shown to improve model predictions and explanations . |
| Approach: | They propose to use machine-augmented human attention supervision to enhance model quality. |
| Outcome: | The proposed method is more effective than existing methods requiring higher annotation cost . the proposed method can be trained to generate similar attention to human supervision . |
Incorporating Zoning Information into Argument Mining from Biomedical Literature (2022.lrec-1)
Copied to clipboard
| Challenge: | Argumentative zoning is a text zonation scheme that is used to segment text into zones that serve distinct functions. |
| Approach: | They propose to use zoning information to incorporate into argument mining tasks . they add zonation labels predicted by an off-the-shelf model to the beginning of each sentence . |
| Outcome: | The proposed models improve argument mining models without additional annotation cost. |
Target-to-Source Augmentation for Aspect Sentiment Triplet Extraction (2023.emnlp-main)
Copied to clipboard
| Challenge: | Aspect Sentiment Triplet Extraction (ASTE) is an important task in sentiment analysis, but data scarcity limits performance of existing methods. |
| Approach: | They propose a target-to-source augmentation approach to alleviate the issue of data scarcity in Aspect Sentiment Triplet Extraction (ASTE) they use fluency and alignment discriminators to provide feedback and use this feedback to optimize the generator. |
| Outcome: | The proposed approach significantly improves the performance of existing methods. |
Distantly-Supervised Dense Retrieval Enables Open-Domain Question Answering without Evidence Annotation (2021.emnlp-main)
Copied to clipboard
| Challenge: | Open-domain question answering uses evidence retrieved from large corpus to answer questions . state-of-the-art approaches require intermediate evidence annotations for training . however, such intermediate annotations are expensive and methods that rely on them cannot transfer to the more common setting . |
| Approach: | They propose an open-domain question answering approach that alternately finds evidence from an up-to-date model and encourages the model to learn the most likely evidence. |
| Outcome: | The proposed approach improves over weak retrievers on multi-hop and single-hop benchmarks without using evidence labels. |
DNA: Denoised Neighborhood Aggregation for Fine-grained Category Discovery (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to learn compact cluster representations from coarsely labeled data are noisy and degrade the quality of learning. |
| Approach: | They propose a framework that encodes semantic structures of data into the embedding space . they retrieve k-nearest neighbors of a query as positive keys to capture similarities . |
| Outcome: | The proposed framework can retrieve more accurate neighbors and outperform state-of-the-art models by a large margin. |
Improving Language Model Reasoning with Self-motivated Learning (2024.lrec-main)
Copied to clipboard
| Challenge: | Large-scale high-quality training data is important for improving the performance of models. |
| Approach: | They propose a framework that motivates the model to automatically generate rationales on existing datasets and improves the performance of reasoning through reinforcement learning. |
| Outcome: | The proposed model outperforms InstructGPT on multiple reasoning datasets and outperformed InstructGPT on other datasets. |
Debate4MATH: Multi-Agent Debate for Fine-Grained Reasoning in Math (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing data annotation methods suffer from high annotation cost and lack of effective automatic validation. |
| Approach: | They propose a Fine-grained Multi-Agent Debate framework and a dataset that prompts multiple agents to debate and then a Multi-agent Debates Reward Model (MRM) to improve its mathematical reasoning capabilities. |
| Outcome: | The proposed model outperforms the state-of-the-art methods by 1.2% and 3.5% on a GSM8K dataset and 45.1% on the MATH dataset. |
Data-efficient Active Learning for Structured Prediction with Partial Annotation and Self-Training (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Structured prediction is a fundamental problem in NLP, wherein the label space consists of complex structured outputs with groups of interdependent variables. |
| Approach: | They propose a partial annotation approach that selects only the most informative sub-structures for annotation and a method that incorporates the current model's automatic predictions as pseudo-labels for un-annotated sub-structurals. |
| Outcome: | The proposed approach reduces annotation cost over strong full annotation baselines under a fair comparison scheme that takes reading time into consideration. |
FreeAL: Towards Human-Free Active Learning in the Era of Large Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Modern machine learning models require a huge collection of precisely labeled data, which can be labor-intensive and time-consuming. |
| Approach: | They propose a collaborative learning framework that interactively distills and filters the task-specific knowledge from LLMs. |
| Outcome: | The proposed framework improves zero-shot performance on eight benchmark datasets without human supervision. |
Paired by the Teacher: Turning Unpaired Data into High-Fidelity Pairs for Low-Resource Text Generation (2025.emnlp-main)
Copied to clipboard
| Challenge: | a low-resource natural language generation task requires a large number of examples to generate outputs and outputs. |
| Approach: | They propose a teacher-student pipeline that synthesizes accurate input–output pairs without human labels or parallel data. |
| Outcome: | The proposed pipeline synthesizes accurate input–output pairs without human labels or parallel data. |
Is ChatGPT the ultimate Data Augmentation Algorithm? (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Recent research has examined the use of ChatGPT for data augmentation, but only in limited contexts. |
| Approach: | They use ChatGPT to create new data with paraphrasing and zero-shot generation to compare it to seven other algorithms. |
| Outcome: | The proposed model performs exceptionally well on some simpler data, but it does not perform better than the other algorithms. |
PolQA: Polish Question Answering Dataset (2024.lrec-main)
Copied to clipboard
| Challenge: | Recent proposed systems for open-domain question answering (OpenQA) require large amounts of training data to achieve state-of-the-art performance. |
| Approach: | They propose an efficient annotation strategy that increases passage retrieval accuracy@10 by 10.55 p.p. while reducing the annotation cost by 82%. |
| Outcome: | The proposed approach increases passage retrieval accuracy @10 by 10.55 p.p. while reducing the annotation cost by 82%. |
PlanE: Meta Planning of Data, Tuning, and Inference for Extractive-based LLMs (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for optimizing LLMs for task-specific tasks are limited due to the sheer volume of data. |
| Approach: | They propose a Planning framework for constructing Extractive-based LLMs called PlanE . they propose 'data decomposition', instruction tuning, prompt inference and a 'Data-Tuning-Inference' planner . |
| Outcome: | The proposed framework improves performance across different datasets and on different dataset. |